The selection of hyperparameters, such as prompt templates in large language models (LLMs), must often strike a balance between reliability and cost. In many cases, structural relationships between the expected reliability levels of the hyperparameters can be inferred from prior information and held-out data -- e.g., longer prompt templates may be more detailed and thus more reliable. However, existing hyperparameter selection methods either do not provide formal reliability guarantees or are unable to incorporate structured knowledge in the hyperparameter space. This paper introduces reliability graph-based Pareto testing (RG-PT), a novel multi-objective hyperparameter selection framework that maintains formal reliability guarantees in terms of false discovery rate (FDR), while accounting for known relationships among hyperparameters via a directed acyclic graph. Edges in the graph reflect expected reliability and cost trade-offs among hyperparameters, which are inferred via the Bradley-Terry (BT) ranking model from prior information and held-out data. Experimental evaluations demonstrate that RG-PT significantly outperforms existing methods such as learn-then-test (LTT) and Pareto testing (PT) through a more efficient exploration of the hyperparameter space.

Moreover, given the ever increasing number of machine learning models being developed, model selection is becoming increasingly important.

Computing the gradient of model hyperparameters, i.e., hypergradient, enables a promising and natural way to solve the hyperparameter optimization task. However, gradient-based methods could lead to suboptimal solutions due to the non-convex nature of optimization in a complex hyperparameter space. In this study, we propose a hyperparameter mutation (HPM) algorithm to explicitly consider a learnable trade-off between using global and local search, where we adopt a population of student models to simultaneously explore the hyperparameter space guided by hypergradient and leverage a teacher model to mutate the underperforming students by exploiting the top ones. The teacher model is implemented with an attention mechanism and is used to learn a mutation schedule for different hyperparameters on the fly. Empirical evidence on synthetic functions is provided to show that HPM outperforms hypergradient significantly. Experiments on two benchmark datasets are also conducted to validate the effectiveness of the proposed HPM algorithm for training deep neural networks compared with several strong baselines.

As hyperparameter tuning becomes increasingly costly at scale, efficient tuning methods are essential. Yet principles for guiding hyperparameter tuning remain limited. In this work, we seek to establish such principles by considering a broad range of hyperparameters, including batch size, learning rate, and weight decay. We identify a phenomenon we call trajectory invariance, where pre-training loss curves, gradient noise, and gradient norm exhibit invariance--closely overlapping--with respect to a quantity that combines learning rate and weight decay. This phenomenon effectively reduces the original two-dimensional hyperparameter space to one dimension, yielding an efficient tuning rule: follow the salient direction revealed by trajectory invariance. Furthermore, we refine previous scaling laws and challenge several existing viewpoints. Overall, our work proposes new principles for efficient tuning and inspires future research on scaling laws.

We provide visual explanation of how embeddings are passed to MLP in Figure 2 and Figure 3. Also, We provide visualisation of target-aware PLE (subsubsection 3.2.2) in Figure 4. Figure 4: Obtaining bins for PLE from decision trees. We used the following datasets: Gesture Phase Prediction (Madeo et al. [27]) Churn Modeling We follow the pointwise approach to learning-to-rank and treat this ranking problem as a regression problem. In this section, we apply the quantile-based piecewise linear encoding (described in subsubsec-tion 3.2.1 to MLP and Transformer on the synthetic GBDT -friendly dataset described in section 5.1 The results are visualized in Figure 5. In this section, we test Fourier features implemented exactly as in Tancik et al. We mostly follow Gorishniy et al. [13] in terms of the tuning, training and evaluation protocols.

Computing the gradient of model hyperparameters, i.e., hypergradient, enables a

All the experiments were conducted under the same conditions in terms of software versions. The feature preprocessing for DL models is described in the main text. The preprocessing is then applied to original features. The remaining notation follows those from the main text. For most experiments, training times can be found in the source code.